Multi-perspective input for computing devices

ABSTRACT

An electronic device, which may be handheld, is provided with a touch sensor and display screen. The display screen may be on a front surface of the device, and the touch sensor may be on a rear surface. Virtual keyboard key contact areas are responsive to user touch inputs, without providing any visual indicia on the display screen indicative of the positions associated with the key contact areas. Key contact area positions can be dynamically adjusted in response to locations on the touch sensitive input surface of user touch inputs.

RELATED APPLICATIONS AND CLAIM OF PRIORITY

This patent application is a continuation of, and claims the benefit of,U.S. patent application Ser. No. 16/405,867 filed on May 7, 2019 andtitled “Multi-Perspective Input For Computing Devices”.

BACKGROUND

As computing devices, especially mobile computing devices, becomeubiquitous in modern life, touch-based input mechanisms have gained vastpopulation as a human-computer interface. With conventional hapticelectronic device input modalities, users select an input devicelocation by directly touching portions of an input device, such as atouch sensitive display panel, at which selection is desired. Forexample, if the user wants to manipulate a graphical object on a touchsensitive display, one or more of the user's finger would touch theparticular locations on the display corresponding to the graphicalobject.

However, one disadvantage of such direct-on-object touch input isblocking the user's view of the objects to be manipulated, when the usertries to select the objects by touching the screen. Thus, in some casesthe touch input signals are not created accurately, especially when theobject on the display is relatively small such that the user's fingercompletely or partially blocks the user's view of the object. Anotherdisadvantage of the direct-on-object touch input is the inefficiency ofinput. Normally, with direct-on-object touch input, it is convenient forat most two fingers to make touch inputs on the display. Such two-fingerinput may limit the speed of user interaction, especially for inputs incharacter typing.

SUMMARY

Methods and apparatuses for providing interaction between a computingdevice and a device user are disclosed. Multiple touch-sensitive and/orpressure-sensitive surfaces may be provided on various external surfacesof a computing device, such as a smartphone, tablet or other type ofhandheld computing device. A touch-sensitive display screen may be setwithin a front surface of the device, while a rear auxiliarytouch-sensitive surface sensor may be positioned on a rear surface ofthe device for identifying one or more points of haptic contact,preferably in at least two dimensions. One or more side surfaces of thedevice may also include a touch-sensitive surface sensor operable toidentify one or more positions of haptic contact along a range ofcontact points in at least one dimension. A microprocessor may receivesignals indicative of haptic contact engagement occurrences andpositions from each of the front display screen the rear auxiliarytouch-sensitive surface sensor and the side auxiliary touch-sensitivesurface sensors. Such signals may then be applied as user interactioninputs to an operating system and/or application software implemented bythe microprocessor. In some embodiments, the side surfaces may be linearsensors, rather than sensors providing haptic contact information withina two-dimensional range. In some embodiments, side sensors on opposingsides may detect pressure, such that a user squeezing the device (e.g.causing a simultaneous increase in pressure applied to opposing sidesensors) may be detected as an interaction input. The rear sensor may beintegrated into a rear display, and/or side sensors may include anilluminating panel with a state of illumination controlled by hapticcontacts with the sensor.

In accordance with another aspect, a touch-sensitive rear surface of ahandheld electronic device may be beneficially utilized as a keyboardfor user input, thereby potentially providing more front display screenspace for application content or the like. The handheld electronicdevice may be held with thumbs on front display and multiple fingersproximate a rear surface of the device. A plurality of haptic targetsmay be activated on the rear touch-sensitive sensor, each haptic targetassociated with a corresponding input event, such as a keyboard key. Thedevice may then detect a contact with a haptic target, and respond to acorresponding input event by the device processor within the handheldelectronic device. During such a mode of operation, providing forrear-surface keyboard input, indicia of haptic targets may beselectively displayed on the front display screen, potentially bysuperimposing the indicia over display content otherwise occupying thedisplay screen, whereby a user may be provided with guidance for use.Such indicia of haptic targets may be triggered in response tosatisfaction of one or more indicia display criteria, which may be atleast in part software-determined based on, e.g., OS state andapplication state, and/or user-determined criteria (e.g. physicalorientation of the device, or finger positioning). The indicia of haptictargets may optionally include a visual indication of rear surface usercontact points, further guiding the user's use of the rear inputmechanism. In yet other embodiments, a rear sensor may be used as atrackpad, to which a user may apply gestures to interact with contentdisplayed on the front display panel (e.g. to move a cursor or selectionarrow).

The multiple input surfaces described herein may be effectively employedto interact with object representations rendered in three or moredimensions in a scene. First, a selected object representation may beselected from amongst multiple object representations rendered in ascene on a display screen (which may be, e.g., directly integrated withthe handheld electronic device, or part of a separate system (e.g.computer workstation) for which the handheld electronic device may beused to interact. The object representation may be selected by using asequence of haptic contacts with multiple surface sensors. A selectionpath may be specified, e.g. by tapping a position on a front display toconfirm an (x, y) coordinate associated with the path, and tapping therear display to confirm the selected (x, y) coordinate for the path endpoint and/or confirming the prior selection. Another haptic contact maybe utilized to select a selection point along the selection path atwhich the desired object representation exists. Once the objectrepresentation is selected, the user may act upon it via further hapticcontacts with two or more of the touch-sensitive sensors. Examples ofsuch actions include rotation and/or translation. The scene renderingperspective may also be altered during interaction by the user.

According to yet another aspect of certain embodiments, contacts with asurface touch-sensor may be dynamically mapped to triggerable events. Inparticular, an event associated with a haptic contact may be based onone or more spatial properties of the haptic contact relative to otherhaptic contacts. Such spatial properties may include a position ofhaptic contact relative to other haptic contacts by the user on aparticular sensor. In particular, triggering an event may includecorrelating a spatial relationship between haptic contacts to a spatialrelationship between selectable elements. Spatial properties that may beused to trigger events and/or select event types from amongst multipleevent types further include a shape of contact area, an orientation ofcontact area and/or a trajectory of contact movement. In someembodiments, dynamic mapping of haptic contacts to trigger events may beperformed by a handheld electronic device microprocessor implementingoperating system software. In other embodiments, dynamic mapping ofhaptic contacts to trigger events may be performed by a handheldelectronic device microprocessor implementing application software.Triggering an event may include correlating a spatial relationshipbetween haptic contacts with one or more touch-sensitive sensors to aspatial relationship between selectable elements rendered on a displayscreen, which may be a display screen disposed within the handheldelectronic device or a display screen within a second computing device(such as a PC, computer workstation or gaming system) that is inoperable communication with the handheld electronic device. Triggeredevents may be selected from amongst a plurality of event types based on,e.g., one or more of: a sequence of haptic contacts with one or moretouch-sensitive sensors (whether implemented as independent sensors orforming a continuous sensor surface wrapping around the handheldelectronic device); manipulation of one or more objects rendered inthree or more dimensions using a sequence of haptic contacts; ordetection of an orientation of the handheld electronic device, such asvia an inertial measurement unit or other orientation sensor.

In accordance with yet other aspects of the disclosure, haptic contactson a touch-sensitive surface sensor may be dynamically grouped. Multiplehaptic contacts may be clustered into contact groups, such as vianearest neighbor analysis or clustering techniques such as density-basedclustering. Each contact may then be associated with a contact group,with the associated contact group potentially utilized to characterizeinput events associated with the haptic contacts. Haptic contacts mayalso be translated spatially relative to a touch-sensitive surfacesensor, based on an angle of orientation associated with the contacts.For example, a reference path (such as a line or curve of best fit) maybe determined for a group of contacts. An angle of orientation of thereference path relative to the electronic device (e.g. thetouch-sensitive surface sensor) may be calculated, and then utilized totranslate contact positions associated with haptic contacts. Multiplehaptic contacts may also be converted to vary an input parameter. Insome embodiments, contact locations may be determined for a referencehaptic contact and a control haptic contact. A reference path on thetouch-sensitive surface sensor can be calculated based on contactlocations of the reference haptic contact and control haptic contact. Anassociated input parameter may then be determined based on a location ofthe control haptic contact relative to the reference path.

Various other objects, features, aspects, and advantages of the presentinvention and embodiments will become more apparent from the followingdetailed description of preferred embodiments, along with theaccompanying drawings in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is front view of a computing device with multi-surface touchinput.

FIG. 2 is a rear view of the computing device.

FIG. 3 is a schematic cross-section A-A of the computing device.

FIG. 4 is a front upper perspective view of the computing device.

FIG. 5 is a rear lower perspective view of the computing device.

FIG. 6 is a schematic block diagram of the computing device.

FIG. 7 is a perspective view of a user holding the computing device formulti-surface text input.

FIG. 8 is a front view of the computing device during a state in whichrear surface input visual cues are not displayed.

FIG. 9 is a front view of the computing device during a state in whichrear surface input visual cues are overlaid onto a front display.

FIG. 10 is a state diagram for controlling the display of rear surfaceinput visual cues on the front display.

FIG. 11 is a perspective view of a user holding the computing device foruse of a rear surface trackpad input.

FIG. 12 is a front view of the computing device displaying visual cuesfor use of a rear surface trackpad input.

FIG. 13A is a perspective view of a user holding the computing devicefor single-handed multi-surface input.

FIG. 13B is a state diagram illustrating user-controlled enabling anddisabling of auxiliary input surfaces.

FIG. 14A is a process for utilizing a computing device with multipletouch-sensitive surface inputs to act on an object or scene rendered inthree or more dimensions.

FIG. 14B is a top plan view of a user holding the computing device tomaneuver objects within a three-dimensional rendering.

FIG. 15 is a top plan view of a user holding the computing device forselection of an object within a three-dimensional scene.

FIG. 16 is a front view of the user and computing device.

FIG. 17 is a rear view of the user and computing device.

FIG. 18 is a first process for selecting an object within athree-dimensional scene.

FIG. 19 is a second process for selecting an object within athree-dimensional scene.

FIG. 20 is a front view of the computing device implementing anapplication receiving multiple relative multi-surface touch inputs.

FIG. 21 is a rear view of a first configuration for providing such touchinputs.

FIG. 22 is a rear view of a second configuration for providing suchtouch inputs.

FIG. 23 is a front view of a computing device implementing anapplication with static control targets.

FIG. 24 is a rear view of the computing device implementing relativecontrol targets on a rear surface touch sensor.

FIG. 25 is a schematic block diagram of the computing device used as amulti-dimensional input device for a separate computing system.

FIG. 26 is a process for grouping haptic contacts on a touch sensor.

FIG. 27 is a schematic illustration of a process for grouping hapticcontacts on a device rear touch sensor.

FIG. 28 is a process for detecting contact orientation and translatingtouch input.

FIG. 29 is a schematic illustration of a process for detecting contactorientation and translating touch input on a device rear touch sensor.

FIG. 30 is schematic illustration of contact orientation and translationapplied independently to multiple contact groups.

FIG. 31 is a schematic illustration of converting multiple hapticcontacts to a single input.

FIG. 32 is a process for converting multiple haptic contacts to variableinput parameters.

FIG. 33 is a schematic illustration of an electronic device utilizingmultiple haptic contacts to derive rotational and linear inputparameters.

DETAILED DESCRIPTION

While this invention is susceptible to embodiment in many differentforms, there are shown in the drawings and will be described in detailherein several specific embodiments, with the understanding that thepresent disclosure is to be considered as an exemplification of theprinciples of the invention to enable any person skilled in the art tomake and use the invention, and is not intended to limit the inventionto the embodiments illustrated.

In existing human-computer interaction mechanisms that are based on useof a touch-sensitive display screen, the dominant mechanism of userinput is haptic, such as user touch on a touch sensitive screen on thefront of a computing device. Software-defined input mechanisms, such asa touch-active virtual keyboard, are typically displayed on the screenin a manner such that, for many device types such as smartphones andtablets, the input mechanisms occupy a large portion of the limitedscreen territory, which is already often constrained for displayingapplication data even without a virtual keyboard occupying a significantportion of the screen. Also, because the majority of user inputstypically come through the screen on the front of a computing devicethat the user may be holding, most of the user's fingers are notpositioned appropriately for inputting data, due to the human hands'physical topology when holding the computing device facing the user. Forexample, common device holding techniques may leave only a user's thumbspositioned against the front device display, with the remaining fingersall contacting the device back side or other surfaces.

Such factors hinder the efficiency of user input, and in many casesbring inconvenience to the handling of and inputting to the device.Haptic user input mechanisms on a device's front display screen may alsoexpose the user's inputs to observation by other people in sight rangeof the computing device, which may be undesirable in some circumstances.

For these and other reasons, it may be desirable to support multiplesurfaces providing touch-based user input on a computing device,particularly a handheld personal electronic device such as a smartphoneor tablet. Input from touch sensors in or on the multiple input surfacesmay be utilized by operating system and/or application software suchthat inputs sources interact and/or communicate with each other, toenable sophisticated, multi-dimensional user input functionalities.

An exemplary device is illustrated in FIGS. 1-5 . FIG. 1 is a front viewof personal electronic device (PED) 100, illustrated schematically.Front surface 102 includes touch-sensitive display screen 110 settherein. PED 100 may further include one or more auxiliary inputsurfaces 120, each responsive to haptic input such as user touch. In theembodiment of FIGS. 1-5 , auxiliary input surfaces 120 are placed onvarious surfaces of PED 100. FIG. 2 is a rear view of smartphone 100,illustrating rear or back side touch-sensitive surface sensor 120A. FIG.3 is cross-section A-A, illustrated schematically, showing front displayscreen 110 and rear touch-sensitive surface sensor 120A. FIG. 3 andcertain other figures may be simplified for illustration. It iscontemplated and understood that references herein to a sensor being“on,” “mounted on” or “set within” a surface of PED 100 shall mean thatthe sensor is able to detect the occurrence of haptic contact anddetermine information concerning the position of that contact relativeto a particular PED surface; in typical embodiments, such sensors may beimplemented as part of a structure that includes protective shielding(e.g. mounted behind one or more layers of glass, plastic or othermaterial) and the sensor need not be directly and externally exposed onthe PED surface.

Other auxiliary haptic input mechanisms, such as touch-sensitive sidesurface sensors, are also implemented in PED 100. FIG. 4 is a top frontperspective view of PED 100, illustrating top surface auxiliarytouch-sensitive surface sensor 120B spanning a top side of PED 100,along with right side auxiliary touch-sensitive surface sensor 120C,spanning a right side of PED 100. FIG. 5 is a bottom rear perspectiveview of PED 100, illustrating left side auxiliary input surface 120E andbottom side auxiliary input surface 120D.

While PED 100 may be illustrated in some embodiments to havetouch-sensitive and/or pressure-sensitive sensors implemented as devicecomponents integrated directly within the PED 100 computing deviceitself, it is also contemplated and understood that in otherembodiments, such surface sensors can be implemented as externalauxiliary components that may be attached to PED 100. An example of suchan implementation is a protective case that may be installed on asmartphone, having touch-sensors built into its back and/or sidesurfaces and in operable communication with inputs to microprocessor 130to pass on information descriptive of haptic contacts received on suchexternal case sensors.

While PED 100 is illustrated in FIGS. 3-5 as a smartphone withconventional tablet-style form factor, it is contemplated and understoodthat in other embodiments, the uses of multiple input surfaces describedherein may be readily applied to different types of devices anddifferent form factors. In some such embodiments, references to therelative position or orientation of surfaces on such devices may varyduring use. For example, in embodiments featuring a clamshell styleelectronic device (such as a hinged or folding phone), portions of a“front” touch-sensitive surface sensor may in fact face rearwards duringsome modes of use. A “rear” touch-sensitive surface sensor may faceinwards when such a device is folded around it. Similarly, a “top” sidesurface sensor may face upwards during some modes of use (e.g. when atablet with horizontal folding centerline is used in an openconfiguration) and downwards in other modes of use (e.g. when a tabletwith horizontal folding centerline is used in a folded configuration).References to surface positioning here are intended to be relative to aparticular mode of use and to distinguish one surface from another, andshould not be deemed to exclude configurations in which relative surfacepositioning may vary across modes of use, unless such an exclusion isexplicitly stated.

FIG. 6 is a schematic block diagram of PED 100. PED 100 includesmicroprocessor 130, implementing software stored in memory 140, such asoperating system 140A and application software 140B. Memory 140 may beor include a non-transitory computer readable storage medium, impressedwith or containing computer program instructions which, when executed onmicroprocessor 130, cause PED 100 to implement methods and operationsdescribed herein. Touch-sensitive primary display 110 interacts withmicroprocessor 130; for example, microprocessor 130 may drive a graphicdisplay panel integrated within display 110, while display 110 maytransmit signals indicative of haptic contact (i.e. touch occurrencesand touch position on the sensor/display 110), to microprocessor 130.Such haptic contact or touch position information may then be applied asa source of user interaction inputs to software implemented bymicroprocessor 130, such as OS 140A and/or applications 140B. Similarly,auxiliary touch-sensitive input sensors 120 also provide signalsindicative of haptic contact (e.g. engagement occurrences and touchposition on the sensor) to microprocessor 130, for implementation asuser interaction inputs for OS 140A and/or applications 140B. Inertialmeasurement unit (IMU) 150 may be utilized (amongst other things) as anorientation sensor, providing information concerning the orientation ofcomputing device 100, such that, amongst other things, OS 140A may, whenimplemented by microprocessor 130, determine whether computing device100 is being held in a landscape or portrait orientation.

While the embodiment illustrated in FIGS. 1-5 provides a primary displayinput surface and five auxiliary input surfaces on other sides of PED100, it is contemplated and understood that in other embodiments, anycombination of one or more auxiliary input surfaces 120 may be utilized,with varying placement. For example, in some embodiments, two inputsurfaces will be provided: a front surface touch-sensitive display, anda rear surface touch sensor.

Touch sensors of the type used on typical front-side touch sensitivedisplays for personal electronic devices typically track one or morespatial properties of haptic contact, such as finger positioning in twodimensions. However, additional or alternative spatial properties may bedetermined (e.g. by OS 140A and/or application 140B), viacharacterization of an area of contact with a touch sensor. Byevaluating an area of contact, additional spatial properties other thana center or intended point of contact may be determined. For example,some touch sensors additionally may detect absolute or relative amountsof pressure applied while touching the sensor (either measuring pressuredirectly, or by evaluating spatial properties such as the shape of acontact caused by deformation of a user's finger over a two-dimensionalplane). Other spatial properties that may be detected by some touchsensors include an orientation of contact area. For instance, a contactarea may be characterized with a length and width. Characterization ofoval or oblong shapes may be utilized to distinguish between a userpressing down directly with a finger tip, versus laying the length of afinger across a sensor to form an elongated contact area. Theorientation of an oval or oblong contact area (i.e. the contact area'smajor axis) in turn may provide an indication of the direction of auser's finger relative to the touch sensor, and thus, the orientation inwhich a user is holding device 100. Yet another spatial property of ahaptic contact that may be characterized in some embodiments is atrajectory of contact movement. A device operating system may analyzethese and other spatial properties of a haptic contact to passinterpreted haptic contact information to an application program as userinput events.

In some embodiments, one or more of auxiliary surface sensors 120 mayinclude a linear sensor, rather than a two-dimensional sensor. A linearsensor may be particularly beneficial for implementation along the sideedges of a generally thin device, such as top input surface 120B, rightinput surface 120C, bottom input surface 120D and left input surface120E, when implemented in a smartphone or tablet embodiment. In somesuch embodiments, with available surface area limiting the extent towhich meaningful two-dimensional input can be detected, linear sensorsmay be more easily used by a user, with less opportunity for unintendedsensor output. For example, a user can slide a finger along an edge ofPED 100 to provide a linear input signal, without worrying whether thefinger is simultaneously drifting towards the front or rear of thedevice. Such a sensor may output an indication of one or more positionsof haptic contact with the sensor along a single dimension, for example,identifying a contact point along a range of contact points extendingalong some or all of a device side surface.

In some embodiments, one or more of auxiliary input sensors 120 mayutilize pressure sensors, in addition to or in lieu of touch positionsensors. For example, in some embodiments of PED 100, it may bedesirable to implement edge-mounted auxiliary sensors 120B, 120C, 120Dand 120E as pressure sensors only, while implementing rear auxiliarysensor 120A as a two-dimensional touch position sensor, and implementingfront display 110 as a pressure-sensitive two-dimensional touch positionsensor. Thus, front display 110 and rear sensor 120A may togetherprovide multiple touch position inputs to microprocessor 130 (and thusOS 140A and application software 140B), while side surface pressuresensors 140B-E may provide inputs to microprocessor 130 enablingdetection of an event when a user squeezes PED 100 (such as whenauxiliary input pressure sensors on opposite sides of PED 100 detect asimultaneous increase in pressure, potentially also exceeding athreshold level).

In some embodiments, auxiliary input surfaces 120 may further includevisual display panels, such as raster image displays similar to frontdisplay 110 which may be implemented using LED, OLED, AMOLED or othercomparable display technologies. In particular, rear input surface 120Amay be particularly beneficially implemented as a touch sensitivedisplay, given the relatively large surface area available. In someembodiments, auxiliary input surface sensors may include other visualdisplay mechanisms, such as an illuminating panel, preferably having avariable state of illumination determined at least in part by hapticcontacts with the sensor. For example, a pressure-sensitive side surfacesensor may illuminate when a threshold level of pressure is applied toit, and/or cease illumination when a level of applied pressure fallsbelow a threshold level, thereby providing visual feedback reinforcingthe pressure-based side surface input detected by PED 100.

In some embodiments, other touch-based input devices may be utilized onone or more auxiliary input surfaces 120, such as buttons, keys orkeyboards, trackpads, and the like.

Various embodiments utilizing multiple surfaces for touch-based inputmay include one or more of: (1) reduced likelihood of touch input errordue to, for example, decreased spatial density of touch input indicia;(2) increased speed and efficiency of typing and other types of touchinput; (3) freeing up precious screen territory on a primary displayscreen, which otherwise may be occupied by on-screen touch keyboard orother software-defined touch input indicia when the active applicationneeds input; (4) avoid blocking the user's view of the data displayed onthe screen; (5) accurate and affirmative touch-based object or textselection; (6) providing new or additional dimensions of touch-sensitiveand/or pressure-sensitive inputs on one or more surfaces of a device,which enable new types of control on displayed and/or displayperspectives objects and/or display environments; (7) providing new oradditional dimensions of touch-sensitive and/or pressure-sensitiveinputs on one or more surfaces of a device, which enable multiple inputmodes, and support natural switching amongst multiple input modes; (8)much better support for single-handed modes of device handling and userinputting; (9) providing privacy protection when authentication input isprovided by typing and/or hand gestures on the rear surface underneaththe device, which is typically not visible to others in the proximity ofa device user, rather than on the front surface of the device which maybe easily visible to others.

Exemplary modes of use for multi-surface touch input may be understoodin connection with the following embodiments:

Ten-Finger Keyboard on the Rear Surface on the Back of a ComputingDevice.

FIG. 7 illustrates one embodiment of a computing device enablingten-finger input by a human with a handheld electronic device,illustrated with a form factor that is generally flat and rectangular inshape. Computing device 100 can be held in the hands of user 710 in alandscape orientation. User 710's thumbs can be used to provide input onthe front surface 110 of computing device 100, while user 710's otherfingers can naturally rest and/or tap on the rear surface 120A of thecomputing device, which may implement a keyboard input mechanism. Insome embodiments, rear surface 120A may provide a touch-sensitivedisplay panel, implementing a software-defined keyboard for user inputand optionally displaying haptic finger targets. In other embodiments,rear surface 120A may provide a touch-sensitive panel surface without adisplay, optionally including printed indicia indicating target inputareas which the user may utilize for guidance of haptic input as needed.In yet other embodiments, physical keys may be provided on rear surface120A.

One potential challenge with use of rear surface 120A for keyboard inputmay be providing users with visual cues to indicate correspondencebetween haptic targets (i.e. touch locations) and a correspondingcharacter or event for input. Even if visual indicia are provided onrear surface 120A (whether permanently printed, or software-definedusing a rear surface display panel), a user 710 may not be able toeasily observe rear surface 120A during use. However, software-definedinput indicia may be selectively displayed on front display panel 110,preferably overlaid or superimposed onto display content otherwise beingshown on display panel 110 by a current application, in order to providea user with visual cues facilitating input on rear surface 120A.Preferably, such visual cues can be displayed on demand, providing auser with guidance when desired to enable targeting of desiredcharacters for entry or other input events, without otherwise disruptingthe user's perception of the primary display content.

FIGS. 8-10 illustrate an exemplary implementation of a variable frontsurface display providing a user with easily-viewable indicia of rearsurface input locations. In particular, FIG. 8 is a front view ofcomputing device 100, having front display 110 rendering applicationdata 820 during normal use. FIG. 9 is a front view of computing device100, upon triggering of a display overlay 830 providing a user withvisual cues as to the significance of various locations on a rear paneltouch surface.

FIG. 10 is a state diagram illustrating the operation of overlay 830.During an initial state S1000, a visual cue trigger is in an OFF oruntriggered state, such that front display visual cue overlay 830 is notdisplayed, as illustrated in FIG. 8 . Optionally, a first trigger mustbe satisfied to place computing device 800 into a mode of operation foraccepting of rear surface input. Such a first trigger, to enable themode of operation, may be comprised of one or more criteria, which mayinclude various combinations of user-determined and/orsoftware-determined criteria. For example, software-determined triggersmay include criteria such as OS 140A enabling a rear surface keyboard,and/or a current application 140B being in a state in which it acceptskeyboard input. An exemplary user-determined trigger criterium mayinclude a combination of sensor states indicative of desired rearkeyboard operation, such as: (1) holding of computing device 100 in aparticular physical orientation (e.g. landscape orientation or portraitorientation), and/or (2) application of a threshold level of pressure orcontact to both top input surface 120B (facing towards the right, in theorientation of FIG. 8 ) and bottom input surface 120C (facing towardsthe left, in the orientation of FIG. 8 ), and/or (3) a particularauxiliary input surface state of user input, such as resting of fourfingers on each side of rear surface touch sensor 120A.

If mode trigger criteria are satisfied, computing device 800 entersstate S1010, in which a rear surface input location overlay may bealternatively displayed or concealed to guide the user in using a rearsurface touch sensor for input. Indicia display criteria may alsoinclude various combinations of software-determined criteria anduser-determined criteria. For example, in some embodiments, when adevice is in a mode of operation enabling rear-surface keyboard input,it may be desirable to display rear input overlay 830 when a userlightly rests four fingers on each side of rear surface input 120A(state S100B, resulting in a display such as that of FIG. 9 ), and stopor conceal the display of rear input overlay 830 when a user rests fewerthan four fingers on each side of rear surface input 120A (state 1010A,returning the device display to that shown in FIG. 8 ).

In some embodiments, display overlay 830 may further include a visualindication of the touch location of each finger on rear surface input120A. For example, a keyboard display may only render the characters atlocations corresponding to the user's fingers on rear surface 120A.Alternatively, a keyboard display may visually highlight or emphasizecharacters corresponding to touch locations of each finger on rearsurface 120A (e.g. by semi-transparent color block, or by changing theappearance of characters corresponding to touch locations such as bybolding or decreasing transparency).

Multi-Finger Gesture-Enabled Trackpad on the Rear Surface on the Back ofa Computing Device:

In this input mode, the user can use the touch-sensing and/orpressure-sensing rear surface 120A on the back of the computing deviceas a trackpad, which accepts all kinds of pointing inputs and gestureinputs, such as pointing, tapping, pressing, swiping, and other gesturesthat involve one or more fingers.

In this input mode, the computing device can be held by the user ineither landscape orientation (as in the input mode above) or portraitorientation. For example, FIG. 11 illustrates a user 1100 holdingcomputing device 100 in a portrait orientation, using rear surface 120Aas a track pad input. Implementation of the trackpad may be analogous tothe rear keyboard mechanism described above, e.g. in the state diagramof FIG. 10 . For example, initial criteria may be required to place thecomputing device into a mode of operation enabling track pad input, suchas enabling of a track pad within OS 140A and activation of track padinput by a current application 140B executing by PED 100. Other criteriamay trigger a front display overlay or guide providing a user withvisual feedback on use of the trackpad input mechanism. FIG. 12illustrates an exemplary overlay 1200 on front display 110 of computingdevice 100, illustrating track pad boundaries 1202 and a user's currentfinger position 1204.

Single-Handed Computing Device Handling and Inputting:

FIG. 13A illustrates another mode of operation for computing device 100,in which a user may operate computing device 100 using a single hand.Conventionally, single-handed use of a touch screen computing devicelimits a user to relatively simple inputs provided with the user'sthumb, such as scrolling, or selection of an on-screen indicia that ispositioned within reach of the user's thumb. However, by implementingauxiliary touch surfaces as described herein, a user may be providedwith greater input capabilities during single-handed operation.

FIG. 13A illustrates an exemplary user 1300 holding computing device 100using a single hand. The user's thumb 1302 continues to be available asa source of touch input on front screen 110. Additionally oralternatively, one or more of user 1300's other fingers maysimultaneously provide touch input to rear surface 120A. For example,one or more of user 1300's other fingers 1304 may operate a touch-basedtrack pad input by contacting rear surface 120A, as described above inconnection with FIGS. 11-12 . This way, computing device 100 may be heldsecurely, while simultaneously providing user 1300 is options foraccurate and affirmative touch-based inputs.

Enabling and Disabling of Auxiliary Input Surfaces

During the course of using a computing device 100, users may grasp thedevice in a variety of different manners. Meanwhile, some modes ofdevice use may not require or benefit from utilization of auxiliaryinput surfaces. In such situations, inadvertent contacts with auxiliaryinput surfaces may result in unintentional or erroneous inputs. Thus, insome circumstances, it may be desirable to enable a user tointeractively enable and disable auxiliary touch input surfaces.

FIG. 13B is a state diagram illustrating operation of computing device100, in accordance with an exemplary embodiment. In state 1350, reartouch-sensitive surface sensor 120A is disabled, such that device 100 isnonresponsive to haptic contacts with sensor 120A other than apredetermined set of contact types that may be utilized to toggle anoperational mode or state of sensor 120A (such as activating theenabling the sensor). In step S1360, computing device 100 detects apredetermined event associated with activation of rear sensor 120A,causing device 100 to transition into state 1351. In state 1351, rearsensor 120A is enabled or fully-active, such that haptic contacts withsensor 120A may be utilized as described elsewhere herein (for example,without limitation, for typing or text entry, for implementation ofdynamic input targets, for selection and/or manipulation ofthree-dimensional or hyper-dimensional objects or scenes, etc.). Withinstate 1351, computing device 100 may monitor rear sensor 120A or otherinputs for an event associated with transition of computing device 100back to state 1350, in which rear sensor 120A is disabled (step S1361).

A variety of events may be utilized to transition computing device 100between states 1350 and 1351 (i.e. to enable or disable rear sensor120A). In some embodiments, predetermined haptic contact types may beutilized. For example, in embodiments having side surface auxiliarysensors (e.g. sensors 120B, 120C, 120D and/or 120E), simultaneousapplication of pressure to side sensors 120B-E on opposing sides ofdevice 100 (i.e. squeezing of device 100 by the user) may be utilized toactivate rear sensor 120A (i.e. transition from state 1350 to 1351); todeactivate rear sensor 120A (i.e. transition from state 1351 to 1350);or both (i.e. to toggle between states 1350 and 1351).

In some embodiments, computing device 100 may detect a predeterminedmanner of holding device 100 in order to activate (and/or deactivate)rear sensor 120A. For example, rear sensor 120A may be activated inresponse to holding of device 100 in a landscape position (detected via,e.g., an orientation sensor such as IMU 150) coupled with placement oftwo thumbs on front touch-sensitive display screen 110 and four fingerson each side of rear surface sensor 120A. In another example, rearsensor 120A may be toggled between states 1350 and 1351 in response tosimultaneous tapping of four fingers on rear sensor 120A.

In other example, the relative position of touch points on rear sensor120A may be utilized as additional criteria for transitioning betweenstates 1350 and 1351; such embodiments may utilize operations describedelsewhere herein as dynamic mapping of touch inputs, in order toidentify inputs triggering changes in state. For example, hapticcontacts arranged relative to one another in a particular shape(regardless of absolute position on sensor 120A) may be utilized totransition from state 1350 to 1351, such as placement of multiplefingers on sensor 120A in a curved shape. Such a shape may be determinedby relative positioning of haptic contacts, and in some embodiments, maynot even be dependent upon a specific number of haptic contacts butrather a predetermined path in which the contacts are made (e.g. hapticcontacts in the shape of a curve, whether the curve is formed from 4, 5or even more points of contact).

In yet other embodiments, criteria for state transition may additionallyor alternatively include detection of a predetermined temporal sequenceof contacts. For example, a user may utilize fingers on one hand toeffect a rapid succession of haptic contacts on rear sensor 120A withrelative positions progressing from left-to-right in order to activaterear sensor 120A (i.e. transition from state 1350 to state 1351, withleft-to-right positioning being determined in view of the physicalorientation of computing device 100 as sensed by IMU 150), and the usermay effect a rapid succession of haptic contacts on rear sensor 120Aprogressing from right-to-left in order to deactivate rear sensor 120A(i.e. transition from state 1351 to state 1350).

In some embodiments, other criteria may additionally or alternatively beutilized to control transitions between states 1350 and 1351. Forexample, OS 140A and/or an application 140B may be required to enter apredetermined mode of operation or trigger a software-enabled event inorder to activate sensor 120A, or enable detection of predeterminedtypes of haptic contacts that may activate sensor 120A.

While FIG. 13B illustrates two states of use for rear sensor 120A, it iscontemplated and understood that in other embodiments, a greater numberof states of operation may be employed, with various combinations ofsoftware-determined and/or haptic contacts utilized to transitionbetween states. For example, a number of different active states may beprovided, in which sensor 120A provides different modes of input orutilization relative to OS 140A and/or an application 140B (e.g.toggling between rear surface text input, track pad sensor, ormanipulation of three-dimensional or hyper-dimensional objects orscenes).

While FIG. 13B and discussion relative thereto illustrates examples oftransition between operating modes for rear sensor 120A, it iscontemplated and understood that in some embodiments, analogousoperations may be utilized to activate, deactivate or otherwisetransition between modes of operation for other auxiliary sensors suchas sensors 120B-E.

Object and Perspective Maneuvers in Three or More Dimensions:

When the user uses multiple fingers on multiple touch-sensing and/orpressure-sensing surfaces of the computing device 100, combined gesturessimultaneously applied on multiple input surfaces can be used to performthree-dimensional or hyperdimensional operations on representations ofobjects rendered in three or more dimensions and/or on the viewperspective or environment in which object representations or scenes arerendered.

In some embodiments, a two-step process may be employed to act onrepresentations of objects rendered in multiple dimensions. Asillustrated in FIG. 14A, an object to be acted upon may first beselected from a scene or environment rendered by PED 100 (e.g. ondisplay 110) through user interaction with multiple touch-sensitivesurface inputs (step S1450). Once an object is selected, the object maybe acted upon in multiple dimensions through further user interactionwith multiple touch-sensitive surface inputs (step S1460). In someembodiments, during step S1450, and/or during step S1460, a user mayalso interactive with touch-sensitive surface inputs to manipulate theperspective or environment in which the scene is rendered.

A selected object may be acted upon to translate the object veryprecisely and efficiently within a three dimensional or hyperdimensionalspace using devices with multiple touch-sensitive surface inputs, asdescribed herein. For example, a 3-D gesture with two fingers (one onback surface 120A; one on front surface 110) moving in approximatelyopposite directions can be used to rotate an object displayed on thefront screen, analogously to as if a physical object is held between thetwo fingers. In another possible 3-D maneuver, a similar 3-D gesture canalso be used to change the viewing angle of a scene displayed on thefront screen 110.

An example of such a rotation gesture is shown in the top plan view ofFIG. 14B. A user's right hand 1400 grasps computing device 100 betweenthumb 1402 (contacting front touch sensitive display screen 110) andindex finger 1404 (contacting rear input surface 120A). Left hand 1410may be used to grasp and better stabilize computing device 100. Torotate an on-screen object clockwise, as viewed from above, index finger1404 may move in direction 1420 on rear surface 120A, while thumb 1402may move in direction 1430 on front display 110. Similarly, to rotate anon-screen object counterclockwise, as viewed from above, index finger1404 may move in direction 1430 on rear surface 120A, while thumb 1402moves in direction 1420 on front display 110. In other modes ofoperation, these gestures may be utilized to rotate a view perspectivein which a scene is rendered on display 110. The on-screen objectrepresentation and/or view perspective may be rotated around an axis ofrotation that is perpendicular to the direction of the movement gesturesof finger 1404 and thumb 1402.

Such rotation operations are one example of a number of differenttranslation that may be applied to an object representation and/or viewperspective, in response to haptic inputs applied simultaneously tomultiple different touch sensors on multiple different device surfaces.For example, haptic contacts and gestures on multiple touch-sensitivesensors applied on multiple device surfaces may be utilized to quicklyand intuitively alter an object position or view perspective within, forexample, any of conventional six degrees of freedom, including changesin position (e.g. forward/backward motion, upward/downward motion, andleft/right motion) and changes in orientation (e.g. yaw, pitch androll).

Multiple touch-sensitive inputs may also be utilized forhyperdimensional translation or manipulation of an object and/or scene.For example, an object rendered in three dimensions may be furtheranimated over time, with time being a fourth dimension of user controlbased on haptic contact with multiple touch-sensitive surfaces. In oneembodiment, a user may hold PED 100 as illustrated in FIG. 14B tomanipulate a selected object representation in three dimensions, whilesimultaneously sliding finger 1401 along lower side surface sensor 120E(not visible in FIG. 14B) to control a time dimension, such that a usermay control a desired translation of an object representation over time.

In another use case, simultaneous application of touch input gestures onmultiple touch surfaces may be used to manipulate objects within athree-dimensional or hyperdimensional space, where the view of theobject may be visually blocked (partially or completely) as rendered tothe user. Such gestures may be particularly beneficial in 3-D designapplications, enabling rapid selection and manipulation of objects incomplex scenes.

For example, when considering selection and manipulation of an objectionin three-dimensional (x, y, z) space, two fingers may be used to definea selection path between starting and ending (x, y) coordinates atminimum and maximum (z) depths rendered in a current scene, with a thirdfinger selecting a (z) coordinate along the selection path for objectselection. Such a mechanism is illustrated in FIGS. 15, 16, 17 and 18 .FIG. 15 is a top view of computing device 100, held in a landscapeorientation between user left hand 1510 and user right hand 1520. FIG.16 is a front view of computing device 100, used as illustrated in FIG.15 . FIG. 17 is a rear view of computing device 100, used as illustratedin FIG. 15 .

FIG. 18 is a process for using a computing device to select an object inthree-dimensional space using multiple touch surfaces. In step S1800, atouch point selected on front display screen 110 (i.e. by thumb 1521 inFIG. 16 ) defines a selection path starting point; specifically, an (x,y) coordinate associated with a selection path, with the selection pathstarting point having a depth (z) corresponding to the closest (i.e.front-most, in the z dimension) object rendered in a scene 1600 ondisplay 110, at the selected (x, y) coordinate. In step S1810, a touchpoint selected on rear touch surface 120A (i.e. by index finger 1522 inFIG. 17 ) confirms the selection path, typically by contacting surface120A at a point directly behind (or close thereto) the point selected instep S1800 on front display 110. Specifically, contact with rear sensor120A defines a selection path end point, having an (x, y) coordinatespecified by the user's selection in step S1810 and a (z) dimension ofthe farthest (i.e. rear-most, in the z dimension) object rendered inscene 1600 on display 110 at the selected (x, y) coordinate. In stepS1820, the contact point of a third finger (in this case, left handindex finger 1512) on rear touch surface 120A specifies a (z) dimensiondepth of an object desired for selection, within the selection pathrange defined by start and end points of steps S1800 and S1810,respectively. Thus, a complex three-dimensional selection may be madevery quickly, if not instantly, using a device having only atwo-dimensional display, even in circumstances where the desiredselection point is obscured to the user when rendered two-dimensionally.

While the process of FIG. 18 is illustrated in a particular sequence ofsteps, it is contemplated and understood that in some embodiments, theorder of steps may be altered. In yet other embodiments, the order ofsteps may be unimportant; for example, a selection path starting pointmay always be defined by the (x, y) coordinate of touch contact withfront display 110, regardless of whether the selection path end point orselection point depth selection have been specified by contacts to reartouch surface 120A.

In step S1820, the desired distance along the selection path defined insteps S1800 and S1810 may be determined by a one-dimensional inputspecification. In some embodiments, the selection point (z) dimensionmay be determined by a relative distance from another finger's contactpoint; for example, the further (z) selection finger 1511 is from (x, y)selection finger 1521 on rear surface 120A, the greater the (z)dimension depth of the associated selection point. In other embodiments,the selection point (z) dimension may be specified by an absoluteposition of a touch point relative to a touch surface sensor. Forexample, in an embodiment having a one-dimensional “slide bar” touchsurface on auxiliary bottom touch surface 120D, the (z) dimension of aselection point may be specified by the relative proportional distanceof a selection point along bottom auxiliary surface 120D. Alternatively,the (z) dimension of a selection point may be determined by a proportionof the distance between the top edge and bottom edge of rear surface120A associated with a touch contact along the left-hand side of rearsurface 120A, such that the user may slide up or down along theleft-hand side of rear surface 120A in order to vary the selectiondepth.

In some embodiments, rather than using two touch points to define aselection path and a third touch point to select a selection depth, suchselections may be made using only two touch points. For example, in theprocess of FIG. 19 , (x, y) coordinates associated with a touch pointselection on front panel 110 (e.g. by thumb 1521 in FIG. 16 ) maydetermine (x, y) coordinates of a desired object selection (step S1900),while a touch point on rear touch surface 120A (e.g. by finger 1522) maydetermine a (z) coordinate of the desired object selection (step S1910).

In step S1820, a number of different techniques may be utilized totranslate the third touch location to a (z) dimension selection depth.In some embodiments, it may be desirable to define a (z) selection rangealong a contour plane over the device surface that connects the two(x,y) points on the front and rear panels selected in steps S1800 andS1810. The closer the (z) selection touch point is to the front surfaceselection point from step S1800, the shallower the (z) selection depth;likewise, the closer the (z) selection touch point is to the rearsurface selection point from step S1810, the deeper the (z) selectiondepth. Thus, the user intuitively selects a depth closer to the “front”of the scene when the (z) selection point is closer to the front display(x, y) selection point, and the user selects a depth closer to the“rear” of the scene when the (z) selection point is closer to the reardisplay (x, y) selection point. In other words, if one defines d1 as theshortest distance over the surfaces of device 100 from the (z) finger tothe (x,y) point on the front screen, and d2 as the shortest distancefrom the (z) finger to the (x,y) point on the rear panel, a (z)selection coordinate may be calculated such that the following tworatios equal: d1/d2=(z1−z)/(z−z2).

Different approaches may additionally or alternatively be implemented tospecify a desired (z) selection coordinate. In an embodiment, a thirdsurface sensor may be used to specify (z) selection depth, such as alinear touch sensor extending along a lower side edge; e.g., a touchposition along a left edge of sensor 120E may correspond to a shallowestscene selection depth, while a touch position along a right edge ofsensor 120E may correspond to a deepest scene selection depth, withinterim depths corresponding to interim touch locations on sensor 120E.

In another embodiment, gestures may be utilized to adjust a currentselection depth, analogous to use of a trackpad for 2D pointernavigation. For example, a haptic contact with rear touch sensor 120A instep S1820 consisting of a swipe towards the rear display (x, y)selection point of step S1810 may cause a current (z) selection depth tomove rearwards by an amount proportional to the length of the swipe;likewise, a swipe away from rear display (x, y) selection point of stepS1810 may cause a current (z) selection depth to move forwards withinthe scene proportionally to the length of the swipe.

In some embodiments, a visual cue may be provided to a user to conveythe current selection point in three-dimensional space during theselection processes described above. For example, the coloration,brightness, contrast and/or other appearance characteristics of anobject closest to the current selection point may be varied on display110 in order to convey to a viewer the object that would be selected ifthe current touch inputs are confirmed. In another example, a pointerobject may be rendered three-dimensionally within scene 1600 as aposition indicated by current selection point. In the depiction of scene1600 in FIG. 16 , three balls are rendered, each with increasing zdimension depth and slightly offset in their x and y positions, suchthat they are rendered as overlapping on front display 110. Middle ball1610 is rendered with bold lines, providing a visual cue that the user'scurrent finger positions would act to select ball 1610.

When the (x,y,z) coordinates of the selection point and/or object to beselected are determined using any of the foregoing techniques, the usercan then provide a confirming input to actually select a target object.Exemplary confirming inputs may include, without limitation, hard-pressor long-press by one or more of the fingers involved in object selection(e.g. thumb 1521 or fingers 1511 and 1522), or conveyance of an audiblecommand by the user (which may be interpreted, for example, by a speechrecognition function implemented by OS 140A).

When an object is selected, operations can be performed on the object byusing the fingers mentioned above, with or without other fingers'further involvement. For instance, a user may move the z finger 1511 tochange the depth of the selected object, while the (x,y) coordinates ofthe selected object can be maintained by keeping the xy selecting finger1521 staying at its selecting position. A user may move the xy selectingfinger 1521 to change the (x,y) location of the object. Moving the frontand back xy fingers 1521 and 1522 in opposite directions may act torotate the object.

Dynamically Mapping Between Touch Inputs and Controlled Objects on theScreen:

In some cases of current touch screen inputs, visual elements on adisplay screen that are responsive to touch input (such as buttons,tools, or other software-defined selectable inputs) are placedstatically on a screen display. The requirement of touching pre-fixedvisual elements statically located at absolute positions on the screenin order to make correct inputs may, for example, be a seriouslimitation when more than two elements need to be controlled at the sametime, because users typically use one or two fingers for touch inputs ona front screen. In such circumstances, users would have to very quicklyswitch amongst multiple fixed input elements. Such static selectableelements may be very inefficient and cumbersome by, for example,consuming screen space even when the element is not used, and obscuringportions of the screen that the user is touching. In some circumstances,static selectable elements may be placed inconveniently along edges of adisplay to reduce overlapping of important application display areas,but as a result, they may be difficult for users to reach. Additionallyor alternatively, selection of such elements may require positioning ofa user's fingers in such a way that the user's view of other elements orinformation of the display is obscured.

In some embodiments, dynamic mapping between touch location and controlmay be utilized, potentially enabling movement of a user's fingers awayfrom the user's sight lines during use. The touching points by theuser's hand may be dynamically mapped to different input elements oractions (whether those input elements or actions are rendered visuallyon a device display screen or not), according to the relative locationof touch points to one other. Dynamic mapping of touch-based inputgestures may be particularly advantageous when implemented with atouchscreen or touch sensor that is positioned on a surface not directlyfacing the user, because in such circumstances a user may otherwise havedifficulty consistently and accurately contacting sensor targets thatare at static locations. Static sensor targets implemented on surfacesor sensors facing away from the user may be particularly challengingwith relatively small or numerous sensor targets; for example, while auser may be able to reliably contact “left half of surface” and “righthalf of surface”, reliably selecting from amongst (for example) four ormore portions of a surface may be challenging when borders betweenregions are static.

Furthermore, the dynamic mapping can be supplemented by other sensors.For instance, data from IMU sensors 150, which detect the orientation ofthe device, can be combined with the relative location information oftouch points to determine a mapping between touching points and theinput elements on the screen.

For example, FIG. 20 illustrates a display on front display screen 110of computing device 100, rendered by an application 140B allowing usersto play a 4-string instrument, including renderings of strings 2000,2010, 2020 and 2030 as selectable elements within the application 140B.FIG. 21 illustrates control of such an application using rear touchsurface 120A, wherein four of a user's fingers 2100, 2110, 2120 and 2130are determined to control actuation of strings 2000, 2010, 2020 and2030, respectively. In particular, haptic contacts from the user'sfingers 2100, 2110, 2120 and 2130 are detected by rear touch-sensitivesensor 120A, and utilized to trigger various events in response to thosehaptic contacts, where each event is selected from amongst a pluralityof event types based upon the position of the haptic contact relative toother haptic contacts. Thus, haptic contacts from fingers 2100-2130 maybe dynamically mapped (e.g. by application software 140B executed byprocessor 130) to events associated with actuation of strings 2000(rendered as selectable indicia on display 110), according to eachfinger's location relative to other fingers as compared to the locationof each string 2000 relative to other strings on the screen 110, ratherthan requiring specific touchpoints on surface 120A to be associatedwith each an actuation event for each string 2000. Thus, when a userholds computing device 100 in the orientation of FIG. 22 , whereinfingers 2100-2130 are distributed horizontally across rear surface 120Arather than vertically, each finger may maintain its allocation tostrings 2000-2030 due to the finger's touch point relative to otherfingers. As a result, the user does not have to touch the one exactplace corresponding to the static location of a string actuation event,in order to manipulate the string. Instead, as long as the four fingershave a relative spatial order amongst themselves when touching the touchsurface (whether front screen 110 or an auxiliary touch surface 120),each finger will be mapped to an event for actuation of the correctstring it is supposed to touch or control. In some embodiments, suchmapping of haptic contacts to software-defined events may beaccomplished by implementation of OS 140A or Application 140B bymicroprocessor 130. While the example of FIGS. 20 and 21 are explainedin the context of dynamic mapping of contacts with rear sensor 120A, itis contemplated and understood that in other applications, a user'scontacts with front display screen 110 may additionally or alternativelybe mapped dynamically to input targets or non-displayed input events.Thus, for example, a user may position fingers 2100, 2110, 2120 and 2130against front screen 110 but in the horizontal orientation shown in FIG.22 , in order to control individual actuation of thevertically-distributed string indicia 2000, 2010, 2020 and 2030,respectively.

Another example is shown in FIGS. 23 and 24 , providing dynamic left andright controls on auxiliary touch surfaces. Conventionally, controltargets may be rendered on front display 110 in fixed positions; forexample, in a racing game application, target 2300 may be associatedwith a braking function, while target 2310 may be associated with anaccelerator function. Targets 2300 and 2310 may occupy screen area thatis obscured by a user's fingers during use, and/or which may beuncomfortable for a user's particular hand size and holding position.However, in computing devices providing multiple touch surfaces fortouch-based input, relative touch controls may be provided on surfacesother than front screen 110, avoiding obscuring portions of screen 110during use of an application and providing users with increasedflexibility in holding position. FIG. 24 illustrates such an embodiment.Rear surface touch sensor 120A may receive user inputs at locations 2400and 2410, from fingers 2420 and 2422, respectively. IMU 150 is anorientation sensor providing data to microprocessor 130 enablingdetermination of the spatial orientation of computing device 100, suchthat computing device 100 may determine the left/right relationshipsbetween locations 2400 and 2410 relative to a user holding computingdevice 100. The leftmost of the touch locations on rear surface 120A maythen be associated with a braking function, with the rightmost of thetouch locations on rear surface 120A may be associated with anaccelerator function.

In some embodiments, dynamic mapping may include a contact groupingfunctionality. Contact grouping may be useful to relate or characterizedifferent points of haptic contact for purposes of interpreting orcharacterizing such contacts as input events. For example, someapplications may wish to distinguish between a user's left hand fingersand a user's right hand fingers on a touch surface, without reliance onarbitrary and fixed physical contact area boundaries, thereby enablinggreater flexibility in how a user holds and contacts an electronicdevice 100. FIG. 26 illustrates a process that may be used for contactgrouping, particularly for identifying groups of four contacts, such asmay be common on a device rear surface when held with thumbs against thefront surface and fingers against the rear surface (as in FIG. 7 ). FIG.27 is an exemplary view of implementation of the process of FIG. 26 ondevice 100.

As described above, electronic device 100 includes rear auxiliary touchsurface sensor 120A. In FIG. 27 , a user grasps device 100 in a positionanalogous to that of FIG. 7 . The user's left hand fingers create hapticcontacts 2710, 2711, 2712 and 2713. The user's right hand fingers createhaptic contacts 2720, 2721, 2722 and 2723. Using a process such as thatof FIG. 26, contacts 2710-2713 may be associated with a contact group2700. Contact groupings may be determined using a process such as thatof FIG. 26 , by a device OS 140A, by an application 140B, and/or byfirmware integrated within a touch sensor or some other component withindevice 100. The contact group identification may then be potentiallyutilized by, e.g., OS 140A and/or application 140B in order to correlatecontact events with desired actions. Similarly, contacts 2720-2723 maybe associated with a contact group 2701.

For example, implementing the process of FIG. 26 , a location on sensor120A is determined for each haptic contact 2710-2713 and 2720-2723 (stepS2600), e.g., such as by microprocessor 130 interacting with auxiliaryinput surface sensor 120A under control of firmware and/or OS 140A todetermine contact positions. For each of those contacts, stepsS2610-S2650 may be performed to determine a number of nearest neighbors.Considering a first reference contact 2711, two closest other contactsare identified (i.e. contacts 2710 and 2712) (step S2610). For a firstclosest contact identified in step S2610 (e.g. for contact 2710), adetermination is made as to whether reference contact 2711 is alsoamongst the two closest contacts. If so, reference contact 2711 isdeemed to include the first closest contact 2710 as a nearest neighbor(step S2630), e.g., a nearest neighbor count for reference contact 2711may be incremented. Next, a determination is made as to whetherreference contact 2711 is also amongst the two closest contacts forreference contact 2711's other closest contact, contact 2712. If so,reference contact 2711 is deemed to include second closest contact 2712as a nearest neighbor (step S2650), e.g. a nearest neighbor count forreference contact 2711 is incremented again. Finally, in step S2660, acontact group may be identified as a series of contacts bounded bycontacts having one nearest neighbor. Implemented in the example of FIG.27 , contacts 2720 and 2723 have one nearest neighbor, while contacts2721 and 2722 have two nearest neighbors, such that contacts 2720 and2723 set the bounds of contact group 2701. Similarly, contacts 2710 and2713 have one nearest neighbor, while contacts 2711 and 2712 have twonearest neighbors, such that contacts 2710 and 2713 set the bounds ofcontact group 2700.

In other embodiments, other techniques for grouping contacts may beutilized. For example, contacts may be grouped using known clusteringtechniques, such as density-based spatial clustering, examples of whichare describe in, inter alia, Ester, Martin; Kriegel, Hans-Peter; Sander,Jörg; Xu, Xiaowei (1996). Simoudis, Evangelos; Han, Jiawei; Fayyad,Usama M., eds., A density-based algorithm for discovering clusters inlarge spatial databases with noise, Proceedings of the SecondInternational Conference on Knowledge Discovery and Data Mining(KDD-96), AAAI Press. pp. 226-231, CiteSeerX 10.1.1.121.9220, ISBN1-57735-004-9.

Determinations of contact groups 2700 and 2701 may thereafter beutilized (e.g. by implementation of OS 140A and/or an application 140Bby microprocessor 130) to characterize input events. For example, apredetermined number of contacts within a contact group may be used totrigger an event, mode of input, or the like. In some embodiments, arange of area on a touch sensor such as rear sensor 120A may be definedaround contacts within a contact group (such as areas 2700 and 2701),such that repeating or additional haptic contacts within the definedareas are also characterized as being associated with a particularcontact group.

In some embodiments, it may additionally or alternatively be desirableto translate dynamic inputs by rotating the x-y coordinate space basedon orientation of the input device and/or based on the relativepositioning of multiple contacts. Such translation may be useful toobtain accurate input from a user, even then the user's mentalconception of the input device orientation in space differs from theactual physical orientation on an input surface.

Orientation detection and translation may be implemented by sensorfirmware, a device OS, or application software. In some embodiments,orientation and translation may be performed using a linear best fittechnique. A linear best fit technique may be particularly effective inapplications where haptic contacts are anticipated over an elongatedarea, such as might be associated with the natural side-by-sidearrangement of a user's fingers. Linear or elongated contact areas mayalso be associated with virtual, software-implemented analogs to manualreal-world devices such as an arrangement of guitar strings spanning thewidth of a guitar fretboard, or a sequence of side-by-side keys on akeyboard. However, an electronic device (particularly a small handheldelectronic device and/or a touch screen device with smooth surfaceslacking significant physical landmarks) may not provide a user with asstrong a sense of physical orientation as a physical guitar, keyboard,or other item of equipment being simulated, thereby making it moredifficult for a user to accurately and consistently orient fingers onthe electronic device for targeted haptic contacts. Also, users may havedifferent preferences in the way that an electronic device is held.Orientation and translation techniques such as those described hereinmay be helpful in accurately and consistently correlating haptic inputswith desired input events, while providing users with flexibility in theway that a device is held or contacted.

FIGS. 28-29 illustrate an exemplary implementation of orientationdetection and input translation. In step S2800, a set of contact pointpositions on a touch sensor is determined, such as points 2900, 2901,2902 and 2903 on touch sensor 120A in the illustration of FIG. 29 . Insome embodiments, the set of contact point positions determined in stepS2800 may be points associated with a particular contact group, such asa contact group identified using the process of FIG. 26 . In step S2810,processor 130 calculates a line of best fit (such as line 2910) throughthe contact points identified in step S2800. In step S2820, processor130 determines an angle α between the line of best fit 2910, and axis2920 having a predetermined orientation relative to device 100 and touchsensor 120A. In step S2830, haptic inputs are spatially translated byangle α. For example, an arrangement of locations on sensor 120Aassociated with software-defined input targets may be rotated by α, suchthat the arrangement of input targets remains oriented with a user'shand position rather than statically oriented relative to device 100.

In some embodiments, orientation detection and translation may beapplied dynamically. For example, the process of FIG. 28 may beperformed each time a user simultaneously applies four fingers to atouch sensor surface, such as rear auxiliary input touch sensor 120A.Such implementations may be useful for allowing a user to periodicallyvary holding position of device 100.

While the embodiment of FIGS. 28-29 illustrates a linear best fitapproach to determining a user's hand orientation relative to device100, it is contemplated and understood that alternative approaches maybe used. For example, in some embodiments, rather than determining alinear best fit in step S2800, best fit of a predetermined curve may bedetermined. In some embodiments, the predetermined curve may be definedbased on a particular user's natural finger orientation, whereby a usermay define the curve used for orientation detection by naturally restingthe user's fingers on a touch sensor such as sensor 120A.

In some embodiments, orientation detection and translation may beapplied separately to different touch sensor surfaces and/or differentportions of a touch sensor surface. For example, in the embodiment ofFIG. 30 , contacts 3001, 3002, 3003 and 3004 may be associated with afirst contact group. The process of FIG. 28 may be applied to thosecontacts to determine an angle β, with angle β applied to hapticcontacts on portion 3020 of sensor 120A. Similarly, a user's other handmay be associated with contacts 3011, 3012, 3013 and 3014, whichcontacts may be utilized in the process of FIG. 28 to determine an angleχ, which angle χ may then be applied to haptic contacts on portion 3030of sensor 120A. In this manner, device 100 may compensate for variationsin holding position for each hand separately.

In some embodiments, dynamic mapping implementations may include aninput logic that produces linear or continuous events based upon therelative distance and/or direction of multiple haptic contacts. Thus,multiple haptic contacts may be converted into a single input parameter,which may be a single discrete event or a continuously varying inputparameter. For example, a device application or operating system mayinterpret a rapid sequence of haptic contacts along a linear path as asingle, discrete input event having a unique characterization includinga direction characteristic. As another example, a device application oroperating system may interpret rapidly-repeating haptic contacts as adiscrete “fast tapping” event (which may be characterized, for example,by a single location characteristic (whether absolute or dynamic) and anaverage tap frequency), rather than interpreting the input as a largenumber of separate haptic contacts.

In another example, two fingers may be utilized to control a pointer orotherwise perform a pointing function (or another function in which aninput parameter is varied based on multiple haptic contacts), withoutregard to the absolute placement of the fingers on a touch sensor. FIGS.31 and 32 illustrate such an embodiment. Two fingers may be placed inpositions 3100 and 3101 on sensor 120A to define initial positions (stepS3200). One of the fingers (e.g. contact position 3100) will preferablyremain relatively static in position on sensor 120A, while contactposition 3101 is translated by a user about contact position 3100. Therelative movement (e.g. in two dimensions) of dynamic contact position3101 relative to anchor contact position 3100 may then be utilized as acorresponding relative change in an associated input event, such asrelative movement of a pointer on primary display 110, or relativedirectional movement of another software-defined indicium rendered ondisplay 110 (step S3210). While reference contact 3100 will preferablyremain relatively static, it is contemplated and understood that in someuse cases, the location of a user's finger associated with hapticcontact point 3100 on surface sensor 120A may drift; in suchcircumstances, the reference by which the relative offset of controlhaptic contact 3101 may be adjusted with movement of contact point 3100,such that the resulting input provided to, e.g., an application layer ofsoftware on device 100, is determined by the relative offset betweenhaptic contacts 3100 and 3101.

Converting of multiple haptic contacts into a discrete orlinearly-varying input parameter may also be effectively utilized forinputs such as speed controls. For example, an initial spacing betweencontacts 3100 and 3101 may define a zero-movement state for an objectrendered on display 110 (such as a race car or the like). Movement ofdynamic contact 3101 further away from anchor contact 3100 may beassociated with an increase in speed. Movement of dynamic contact 3101closer to anchor contact 3100 may be associated with a decrease inspeed.

In some embodiments, converting operations as described herein mayinclude both a directional and speed component. For example, withcontact 3100 defining a reference point, a change in angle of rotationof contact 3101 relative to anchor contact 3100 may be associated with adirectional input, which a change in distance between contacts 3100 and3101 may be associated with a change in speed. Thus, for a running game,users may dynamically control both direction and speed with a singlepair of haptic contacts, on any of multiple touch sensitive screens ortouch sensitive input sensors. Similarly, for a first person shooter, apair of haptic contacts may be utilized to control gun operation, withdirectional input utilized for weapon pointing and speed (i.e. relativedistance) used to control rate of fire or some other aspect of aplayer's operation.

Converting of multiple haptic contacts, and/or patterns of hapticcontacts, into a single event (whether discrete or continuous), may beperformed in a variety of contexts, including by one or more of touchsensor firmware, an operating system (e.g. drivers or libraries), or anapplication program.

In another application, implementations of dynamic mapping may be usedto implement the complex and simultaneous controls in a driving orflying game. For example, in such a game, a vehicle may be rendered bymicroprocessor 130 implementing an application 140B, with the renderedvehicle being controlled by haptic contacts with screen 110 andauxiliary input surface sensors 120. FIG. 33 illustrates an exemplarydynamic control scheme using rear auxiliary input sensor 120A, with bothrotational and linear control references. A first pair of contact pointswith rear sensor 120A, consisting of contacts 3300 and 3301A, may beutilized for changing direction of a rendered vehicle's movement,analogous to turning a steering wheel, using rotational controlreferences. In particular, initiation of haptic contacts 3300 and 3301Adefines reference positions, with reference line 3310 determined asinterconnecting contact points 3300 and 3301A and utilized as a zerorotation, or neutral, reference. Subsequently, a user may slide a fingerassociated with contact point 3301A over touch sensor 120A to, forexample, contact point 3301B. A new rotational line 3313 interconnectspoints 3300 and 3301B, providing a rotational input consisting of anangle ϕ between reference line 3310 and rotational input line 3313.Thus, for example, contact point 3301B may be moved in direction 3311 toprovide input interpreted by microprocessors 130 and a racing gameapplication 140B as steering input towards the right side of a vehiclerendered on front display 110; or in direction 3312 to provide steeringinput towards the left side of a vehicle rendered on front display 110.

Linear (rather than rotational) inputs may be implemented in a similarmanner. For example, in the example of FIG. 33 , haptic contacts 3305and 3306 may be utilized to provide linear input to control rendering ofa race car accelerator or brake position. Initial placement of referencecontact 3305 and variable contact 3306 dynamically defines linearreference path 3320. The distance of contact 3306 from reference path3320 provides a linear input for interpretation by software application140B. Thus, placement of contact position 3306 on reference path 3320provides a zero input, which may be interpreted as neither applicationof accelerator or braking. Movement of contact 3306 in direction 3321may be interpreted as application of an accelerator, with a level ofacceleration determined by a distance in direction 3321 between contactpoint 3306 and reference path 3320. Similarly, movement of contact 3306in direction 3322 may be interpreted as application of braking, with anamount or intensity of braking (analogous to pressure on a conventionalbrake pedal) determined by a distance in direction 3322 between contactpoint 3306 and reference path 3320. While reference paths 3310 and 3320are illustrated in the embodiment of FIG. 33 as linear paths, it iscontemplated and understood that reference paths having curved or othershapes may also be used.

In some embodiments, it is contemplated and understood that a user'shand position may drift during use. Thus, reference paths 3310 and 3320may vary dynamically on sensor 120A, based on the position of referencehaptic contacts 3300 and 3305, respectively. Specifically, for example,linear reference path 3310 may maintain its angle relative to sensor120A but may be translated in X and Y dimensions on sensor 120A based onmovement of reference haptic contact 3300, such that linear path 3310remains passing through contact point 3300. Reference path 3320 may besimilarly translated in X and Y dimensions on sensor 120A, maintainingits angle relative to the sensor, such that path 3320 remains passingthrough reference contact point 3305 despite drifting of the position ofreference haptic contact 3305.

At any point, users may optionally be able to lift and reset contactpair 3305 and 3306, and/or contact pair 3300 and 3301, in order to resetreference paths 3310 and 3320, respectively, (e.g. resetting both X-Yposition and angle) thereby allowing a user to quickly respond tovarying ways of holding device 100 without interrupting deviceoperation. Thus, when implemented on a rear surface touch sensor 120A,the control mechanism of FIG. 33 may provide intuitive control of bothrotational steering input and linear brake/accelerator application usingtwo pairs of fingers, while holding device 100 in a natural position andwithout at all obstructing a user's view of front display screen 110.

In yet another application, a combination of dynamic input mapping,contacts grouping, orientation detection, and input translation can beused to implement a dynamically-floating virtual (computer) keyboard onthe rear or the front surface of a portable or mobile device such asdevice 100. In this case, the user is able to tap on a touch-sensitivesurface of the device for text typing, while not having to rely on astatic position mapping of all the keys as in certain embodimentsdescribed elsewhere herein. Instead, four fingers of the left hand canbe detected as group one; four fingers of the right hand can be detectedas group two. Every time all four fingers of a group make contacts withthe touch-sensitive surface, the four contacts reset the positions oftouch screen target areas corresponding to four predetermined referencekeys. Other key target areas can then be determined relative to thedetected positions of the four predetermined reference keys, with targetpositions and haptic contacts translated based on the detectedorientation of the prior reset positions. In some embodiments, updatedhaptic contact target mapping overlays may be shown in display 110 toprovide a user with additional guidance, analogous to those of FIG. 9but with position and orientation of the contact guide overlaydetermined dynamically as described in this paragraph rather thandetermined statically.

A number of other specific application types may also benefit from oneor more of the multi-surface and/or dynamic touch input mechanismsdescribed herein. One example is applications on mobile devices thatrequire more precise control than fingertip touch selection. Forexample, this type of mobile application may include a photo and/orimage editing application, a 2-D or 3-D object maneuvering/posingapplication, and/or a presentation slides editing tool application.

Another example is graphical game applications on mobile devices. Forexample, real-time strategy games may often display a lot of relativelysmall objects on a crowded display screen. The game play often requiresselecting, moving, acting, etc. on certain objects on the screen. TheMulti-Surface Input system is perfectly suited for these types ofprecision-demanding in-game operations.

Another example scenario in graphical game is the input to first-personand/or third person shooting or fighting games. Multi-surface inputsystems may allow inputs from the user to control the shooting andfighting actions to be applied on the back of the computing device, sothat the control inputs from the user will not block any view of theuser on the front screen, enabling greater information display and amore immersive user experience. Furthermore, multi-surface inputmechanisms can provide 2-D and/or 3-D gestures for additional dimensionsof input, which, for instance, can be used to control and/or change theviewing perspective of the game, besides controlling the shooting and/orfighting actions.

Yet another example use scenario is typing intensive applications onmobile devices. For example, frequently travelling business people,sales people, marketing people, product-oriented people, entrepreneurs,field technicians and/or field operators, maintenance and/or operationengineering people, workers with lengthy commutes, and others may needto enter a large volume of information on a mobile device via typing,such as while writing emails, sending messages, composing documents, andthe like. Multi-surface touch input implementations described herein mayenable text input with improved accuracy and efficiency.

In accordance with yet another use case, a mobile device withmulti-surface input capability may be used as a peripheral input devicefor another computing device. For example, a user of a professional CADsoftware application on a workstation computer may use a mobile devicewith multi-surface input functionality as an input device. Such anarrangement is illustrated schematically in FIG. 25 . Computing device100 may exchange information with workstation 2500 via communicationslink 2550, which may be implemented by, e.g., Bluetooth wirelesscommunications, wireless ethernet, or a cabled data connection. Suchinformation may include, for example, transmission of datacharacterizing haptic inputs detected by device 100, such as touch inputpositions and pressure levels on each of multiple touch input sensors110 and/or 120. Such inputs may be processed by workstation operatingsystem 2510, and conveyed to a software application 2520 for use inapplication control. Thus, computing device 100 may act, in somecircumstances, analogously to a three-dimensional input mouse increating, editing, and manipulating three-dimensional objects in adrawing, painting or design application. Such inputs may be provided toworkstation 2500 in addition to other, conventional control inputs, suchas mouse 2560, keyboard 2562 or touch pad 2564.

Similar to such use of a mobile device as a controller peripheral for aseparate computing device, concepts described herein in connection withmobile device embodiments, may also be utilized in connection withperipheral controller devices that are not otherwise mobile devices orwhich do not otherwise provide standalone computing functionality. Forexample, a controller peripheral with one or more touch-sensitivesurfaces may be implemented using the arrangement of FIG. 25 , in whichcommunications link 2550 enables interaction between workstation 2500(or in some embodiments, a gaming system or other type of computingdevice) and a dedicated controller peripheral device. Such a dedicatedcontroller peripheral may, in some embodiments, be implemented usinghardware components substantially similar to mobile device 100, althoughcomponent specifications may be reduced for cost purposes, includingutilization of a front side touch-sensitive surface sensor in lieu of atouch-sensitive display screen 110.

While certain embodiments of the invention have been described herein indetail for purposes of clarity and understanding, the foregoingdescription and Figures merely explain and illustrate the presentinvention and the present invention is not limited thereto. It will beappreciated that those skilled in the art, having the present disclosurebefore them, will be able to make modifications and variations to thatdisclosed herein without departing from the scope of any appendedclaims.

The invention claimed is:
 1. A handheld electronic device comprising: adisplay screen positioned on a front surface of the electronic device; atouch sensitive input surface detecting user touch inputs; and amicroprocessor receiving signals from the touch sensitive input surfaceindicative of user touch inputs, and driving a visual display on thedisplay screen, the microprocessor configured to: implement a pluralityof virtual keyboard key contact areas on the touch sensitive inputsurface that are responsive to user touch inputs, without providing anyvisual indicia on the display screen indicative of a position associatedwith each key contact area, each key contact area position correspondingto a portion of the touch sensitive input surface dynamically determinedby the electronic device; and dynamically adjust the key contact areapositions in response to locations on the touch sensitive input surfacecorresponding to a plurality of touch input, in which the microprocessoris further configured to: detect a plurality of simultaneous touchinputs on the touch sensitive input surface; and reset the key contactarea positions based on spatial properties of the plurality ofsimultaneous touch inputs, the spatial properties comprising orientationof the plurality of simultaneous touch inputs relative to one another.2. The handheld electronic device of claim 1, wherein the spatialproperties further comprise distances between the plurality ofsimultaneous touch inputs.
 3. The handheld electronic device of claim 1,wherein the spatial properties further comprise a shape of contact areasassociated with the touch input.
 4. The handheld electronic device ofclaim 1, wherein the microprocessor is further configured to, whileresetting the key contact area positions, temporarily display on thedisplay screen visual indicia indicative of a position associated witheach key contact area.
 5. The handheld electronic device of claim 4,wherein the microprocessor is further configured to superimpose saidvisual indicia over display content on the display screen.
 6. Thehandheld electronic device of claim 1, wherein the touch sensitive inputsurface is positioned on a back surface of the handheld electronicdevice.
 7. The handheld electronic device of claim 1, wherein themicroprocessor is further configured to selectively display on thedisplay screen visual indicia indicative of a position associated witheach key contact area.
 8. The handheld electronic device of claim 7,wherein said visual indicative are displayed in response to satisfactionof one or more indicia display criteria.
 9. A method for an electronicdevice to implement a virtual keyboard, wherein the electronic devicecomprises a touch sensitive input surface and one or more displayscreens, the method comprising: providing a plurality of virtualkeyboard key contact areas responsive to user touch inputs withoutproviding any visual indicia on the one or more display screensindicative of a position associated with each key contact area, each keycontact area position corresponding to a portion of the touch sensitiveinput surface dynamically determined by the electronic device; anddynamically adjusting the key contact area positions in response tolocations on the touch sensitive input surface corresponding to aplurality of touch inputs, in which the step of dynamically adjustingthe key contact area positions comprises: detecting a plurality ofsimultaneous touch inputs on the touch sensitive input surface; andresetting the key contact area positions based on spatial properties ofthe plurality of simultaneous touch inputs, the spatial propertiescomprising orientation of the plurality of simultaneous touch inputsrelative to one another.
 10. The method of claim 9, wherein the spatialproperties further comprise distances between the plurality ofsimultaneous touch inputs.
 11. The method of claim 9, wherein thespatial properties further comprise a shape of contact areas associatedwith the touch inputs.
 12. The method of claim 9, in which the step ofresetting the key contact area positions comprises temporarilydisplaying on the one or more display screens visual indicia indicativeof a position associated with each key contact area.
 13. The method ofclaim 12, in which the step of temporarily displaying indicia comprises:superimposing said indicia over display content on one or more of saiddisplay screens.
 14. The method of claim 9, wherein: the electronicdevice is a handheld electronic device; the touch sensitive inputsurface is on a back side of the handheld electronic device; and atleast one of the display screens is on a front side of the handheldelectronic device.
 15. The method of claim 9, further comprising thestep of: selectively displaying on the one or more display screensvisual indicia indicative of a position associated with each key contactarea.
 16. The method of claim 15, in which the step of selectivelydisplaying visual indicia comprises: triggering display of said indiciain response to satisfaction of one or more indicia display criteria.